Game-theoretic Management of Interacting Adaptive Systems

A powerful technique for optimizing an evolving system “agent” is co-evolution, in which one evolves the agent’s environment at the same time that one evolves the agent. Here we consider such co-evolution when there is more than one agent, and the agents interact with one another. By definition, the environment of such a set of agents defines a non-cooperative game they are playing. So in this setting co-evolution means using a “manager” to adaptively change the game the agents play, in such a way that their resultant behavior optimizes a utility function of the manager. We introduce a fixed-point technique for doing this, and illustrate it on computer experiments. I. I Many distributed systems involve multiple goal-seeking agents. Often the interactions of those agents can be modeled as a noncooperative game where the agents are identified with the players of the game and their goals are identified with the associated utility functions of the players [7], [8], [12], [17], [21]. Examples involving purely artificial players include distributed adaptive control, distributed reinforcement learning (e.g., such systems involving multiple autonomous adaptive rovers on Mars or multiple adaptive telecommunications routers), and more generally multi-agent systems involving adaptive agents [2], [6], [15], [16]. In other examples some of the agents / players are human beings. Examples include air-traffic management [9], multi-disciplinary optimization [4], [5], and sense, much of mechanism design, including in particular design of auctions [8], [12], [13]. Sometimes the goals of the players do not conflict; intuitively, the system is “modularized”. However often this cannot be guaranteed. As an example, it may be that in most conditions the system is modularized, but that some conditions cause conflicts among the needs of the players for system-wide resources (e.g., when an “emergency” occurs). Alternatively, it may be that the players take physical actions and that under some conditions the laws of physics couple those actions in way that makes their goals conflict. Moreover, whenever some agents are humans, in almost all conditions there will be some degree of conflict among their goals. Finally, note that even when there are no conflicts among the goals of the players, there may be synergies among the players that they can not readily find if left on their own. In all of these scenarios there is a need to intervene in the behavior of the players. However often we do not have complete control over the behavior of the players. That is always true if there are communication restrictions that prevent us from having full and continual access to all of the players. It is also always true if some of the players are humans. Such limitations onour control are also typical when the players are software programs written by third party vendors. On the other hand, even when we cannot completely control the players, often we can set / modify some aspects of the game among the players. As examples, we might have a manager external to the players who can modify their utility functions (e.g., by providing them with incentives), the communication sequence among them, what they communicate with one another, the command structure relating them, how their chosen moves are mapped into the physical world, how (if at all) their sensory inputs are distorted, or even how rational they are. The ultimate goal of the manager is to make such modifications that induce behavior of the players that is optimal as far as the manager is concerned. As an example, say some of the players are evolvable software systems. Then the game details comprise the environment in which those systems evolve. So modifying the game details to optimize the resultant behavior of the players is a variant of using co-evolution for optimization [1], [3]. The difference with most work on co-evolution is that in optimal management of a game we are concerned with the environment of multiple interacting and evolving systems rather than the environment of a solitary evolving system. In the next section we present a simple example that illustrates how the behavior of players in a game can be improved by changing the game, discuss previous related work, and overview our approach. After that we present our notation. We then use that notation to introduce a formal framework for analyzing optimal management. Next we use our framework to introduce an algorithm for optimal management. After that we present a computer-based test validating that algorithm. II. B To illustrate how changing the details of a game may result in the players behaving in a way that is better for an external manager, say we have two players, Row and Col, each of who has four possible moves (also called “pure strategies”). As usual, each player has a utility function that maps the joint pure strategy of the two players into the reals. Say that those utility functions, (gR, gC), are given by the following bimatrix: